Hagedorn et al. Addiction Science & Clinical Practice 2014, 9:12 
http://www.ascpjournal.Org/content/9/1/12 




ADDICTION SCIENCE & 
CLINICAL PRACTICE 



RESEARCH Open Access 



An implementation-focused process evaluation of 
an incentive intervention effectiveness trial in 
substance use disorders clinics at two Veterans 
Health Administration medical centers 

Hildi J Hagedorn 1 ' 2 ' 3 *, Cheryl B Stetler 4 , Ann Bangerter 2 , Siamak Noorbaloochi 2,3 , Maxine L Stitzer 5 
and Daniel Kivlahan 6,7 



Abstract 

Background: One of the pressing concerns in health care today is the slow rate at which promising interventions, 
supported by research evidence, move into clinical practice. One potential way to speed this process is to conduct 
hybrid studies that simultaneously combine the collection of effectiveness and implementation relevant data. This 
paper presents implementation relevant data collected during a randomized effectiveness trial of an abstinence 
incentive intervention conducted in substance use disorders treatment clinics at two Veterans Health Administration 
(VHA) medical centers. 

Methods: Participants included patients entering substance use disorders treatment with diagnoses of alcohol 
dependence and/or stimulant dependence that enrolled in the randomized trial, were assigned to the intervention 
arm, and completed a post intervention survey (n = 147). All staff and leadership from the participating clinics were 
eligible to participate. A descriptive process evaluation was used, focused on participant perceptions and contextual/ 
feasibility issues. Data collection was guided by the RE-AIM and PARIHS implementation frameworks. Data collection 
methods included chart review, intervention cost tracking, patient and staff surveys, and qualitative interviews with staff 
and administrators. 

Results: Results indicated that patients, staff and administrators held generally positive attitudes toward the incentive 
intervention. However, staff and administrators identified substantial barriers to routine implementation. Despite the 
documented low cost and modest staff time required for implementation of the intervention, securing funding for the 
incentives and freeing up any staff time for intervention administration were identified as primary barriers. 
(Continued on next page) 
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(Continued from previous page) 

Conclusions: Recommendations to facilitate implementation are presented. Recommendations include: 1) solicit 
explicit support from the highest levels of the organization through, for example, performance measures or clinical 
practice guideline recommendations; 2) adopt the intervention incrementally starting within a specific treatment track 
or clinic to reduce staff and funding burden until local evidence of effectiveness and feasibility is available to support 
spread; and 3) educate staff about the process, goals, and value/effectiveness of the intervention and engage them in 
implementation planning from the start to enhance investment in the intervention. 

Keywords: Substance use disorders treatment, Abstinence incentive intervention, Implementation, Process evaluation, 
Hybrid design 



Introduction 

One of the most pressing concerns in healthcare today is 
the slow rate at which promising clinical interventions, 
supported by research evidence, move into clinical prac- 
tice [1-6]. While effectiveness trials are a step in the re- 
search pipeline designed to determine whether promising 
clinical interventions work when tested in "real world" set- 
tings, the results of such trials generally do not provide cli- 
nicians, administrators, or quality improvement specialists 
with the information that they will need to successfully 
implement the new practice in their clinic or hospital 
[7-10]. This is the role of implementation trials which at- 
tempt to develop successful strategies for implementing 
new, effective practices, which then can be disseminated/ 
spread to other clinical settings [8,11]. 

Unfortunately, the timeline from a promising efficacy 
trial, through effectiveness and implementation trials, and 
finally to dissemination can be measured in multiple years 
if not decades and adds to the slow rate of routine uptake 
of promising clinical interventions [4,12-14]. Methods to 
speed this process are needed. One recently proposed 
method is to integrate effectiveness and implementation 
study methods using a hybrid design [15]. Curran and col- 
leagues propose three hybrid types varying in the degree to 
which they focus on effectiveness data versus implementa- 
tion data (See Table 1). The Hybrid Type I encourages re- 
searchers to initiate the study of implementation as early 
as possible in a program of research; more specifically, this 
design focuses on integrating implementation research into 



efforts primarily focused on collecting effectiveness data. 
As described by Curran and colleagues, the additional in- 
corporation of process evaluation methods into an effect- 
iveness trial design can help explain effectiveness results 
while at the same time efficiently informing future imple- 
mentation efforts. In contrast to process evaluation, as de- 
fined in this study as part of a Hybrid I design, formative 
evaluation [9] is used in a Hybrid II or III design to gather 
information for monitoring and actively enhancing imple- 
mentation during the course of the implementation- 
focused aspect of the study. 

Traditionally, process evaluations are used when test- 
ing complex clinical interventions and involve collect- 
ing data from patients and providers to assess issues 
such as whether the clinical intervention was delivered 
as designed, whether the patients received the planned 
"dose" of the clinical intervention, and whether pro- 
viders maintained fidelity to the clinical intervention 
protocol [16-18]. Implementation questions in Hybrid 
Type I studies instead focus more prominently on bar- 
riers and facilitators to uptake/adoption efforts, feasibil- 
ity of the clinical intervention, acceptability of the 
clinical intervention to providers and patients, and on 
identifying tools and training that would assist with 
high fidelity implementation. Therefore, information is 
collected not only from patients and providers but from 
clinical and organizational leaders that would be re- 
sponsible for guiding implementation of the clinical 
intervention. 



Table 1 Typology of hybrid designs* 


Research questions 


Hybrid type 1 


Hybrid type 2 


Hybrid type 3 


Primary 


Effectiveness: Will the clinical 
treatment work in this setting/with 
these patients? 


Effectiveness: Will the clinical 
treatment work in this setting/with 
these patients? 

AND 

Implementation: Does the 
implementation method show 
promise? 


Implementation: Which method works 
better in facilitating implementation of the 
clinical treatment? 


Secondary 


Implementation: What are the potential 
barriers/facilitators to the treatment's 
implementation? 




Effectiveness: Is a clinical treatment 
effective in this setting/with these 
patients? 



*From Curran, Bauer, Mittman, Pyne & Stetler, [15]. 
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Process evaluations frequently involve mixed-methods, 
combining data collected through qualitative methods 
such as interviews, focus groups, and field notes with 
data collected through quantitative methods such as sur- 
veys and administrative data. While there has been a call 
for hybrid studies integrating quantitative effectiveness 
data with quantitative and qualitative implementation- 
focused process evaluation data, there are few published 
protocols documenting the design, execution, and results 
of such trials. This paper describes an implementation- 
focused process evaluation that was tied to a random- 
ized, controlled effectiveness trial of an abstinence incentive 
intervention for substance use disorders [19]. The current 
paper reports how the process evaluation was developed 
and carried out, results of the evaluation, and discussion 
of how these results may inform future implementation 
efforts. 

Methods 

Participants 

This study was conducted at two VHA medical centers. 
At one medical center, participants were recruited from 
the outpatient substance use disorders treatment clinic 
and a dual diagnosis partial hospitalization clinic. At the 
second medical center, participants were recruited from 
the outpatient substance use disorders treatment clinic. 
As approved by the institutional review boards at each 
facility, patient participants completed written informed 
consent and staff participants completed verbal informed 
consent. Patients were eligible to participate if they were 
beginning a new treatment episode for alcohol depend- 
ence and/or stimulant dependence. All clinical staff and 
clinic leaders were eligible to participate. Clinical staff 
was recruited through announcements at clinical staff 
meetings and emails. Clinic leaders were recruited 
through email and telephone contacts. During the study, 
patients were randomized to receive usual care sub- 
stance use disorders treatment with or without financial 
abstinence incentives. All participants were scheduled to 
submit alcohol breath and drug urine screens twice per 
week for eight weeks. Those in the incentive condition 
could draw from a prize bowl when they submitted 
negative breathalyzer and urine screens. The prize bowl 
contained slips that either said "Good Job" but had no 
monetary value or were worth $1, $20, or $80 vouchers, 
which could be redeemed at a VHA cafeteria or gift 
shop. The number of drawings earned increased with 
each consecutive week of abstinence. 

Process evaluation plan 

The initial step in development of the process evaluation 
plan was the selection of an implementation framework 
that would guide the questions asked. For the purpose of 
this study, our team selected the RE-AIM framework 



[20-22] as the primary guide supplemented with ele- 
ments from the Promoting Action on Research Imple- 
mentation in Health Services (PARIHS) framework 
[23-25]. RE- AIM is an acronym for reach, effectiveness, 
adoption, implementation and maintenance. Briefly, 
Reach refers to the percentage of eligible patients that 
agree to the clinical intervention. Effectiveness evaluates 
whether the clinical intervention achieved the desired 
patient outcomes. Adoption refers to the decision of the 
leadership and staff at the targeted settings to implement 
the clinical intervention. Implementation refers to whether 
the clinical intervention is implemented with high fidelity 
and consistency. Implementation also includes an assess- 
ment of the cost of the intervention. Finally, Maintenance 
refers to whether the clinical intervention is sustained 
after implementation support is withdrawn. The frame- 
work proposes that each of these elements is essential to 
consider when planning an implementation strategy or an 
implementation evaluation. According to Kessler and col- 
leagues [26], RE- AIM has been used to "plan, evaluate and 
review a variety of health promotion and disease manage- 
ment interventions" and "over 150 published studies... 
state they are using the RE- AIM model." However, many 
of these studies did not include a comprehensive evalu- 
ation of all framework elements. See Kessler and col- 
leagues' paper [26] on employing the RE- AIM framework 
for an excellent description of general questions prompted 
by each element of the model. 

Guided by the framework, we developed a series of 
questions that would provide valuable information for 
future implementation efforts. (See Table 2, which also 
includes measurement tools discussed below.) Because 
our purpose was to collect information during an effect- 
iveness trial to inform future implementation rather than 
to evaluate an actual implementation effort, our ques- 
tions are slightly different from those recommended by 
Kessler [26]. 

The PARIHS framework proposes that successful im- 
plementation is a function of evidence, context and fa- 
cilitation. For our purposes, we chose to evaluate the 
elements of evidence and context as follows: 

Evidence: In the PARIHS framework, this element 
refers to the strength and extent of evidence for clinical 
practice changes and includes research evidence, 
evidence from clinical practice, perceived patient 
needs and preferences, and local data/information. 
The RE-AIM framework lacks a component specifically 
addressing clinicians' assessments of the evidence 
supporting an innovation. 

Context: In the PARIHS framework, this element refers 
to "the environment or setting in which people receive 
healthcare services" or, in the context of getting 
research evidence into practice, 'the environment or 



Hagedorn et al. Addiction Science & Clinical Practice 2014, 9:12 
http://www.ascpjournal.Org/content/9/1 /1 2 



Page 4 of 1 5 



Table 2 Implementation framework elements guide process evaluation questions and tools 



Element 


Questions 


Data sources 


Tools 


RE-AIM 








Reach 


What percentage of patients approached 
agree to participate in the intervention? 


Recruitment rates. 


Patient screening database. 




Do those that agree to participate differ 


Demographics of those agreeing 


Chart review of administrative data. 




systematically from those that do not? 


vs. those refusing participation. 






What do patients like and dislike about 


Perceptions of patients. 


Post-intervention patient surveys. 




the intervention? 






Effectiveness 3 


What is the effect of the intervention on 


Main study outcomes comparing 


Rates of negative urine screens. 




patient outcomes? 


control to intervention patients. 


Study retention. 


Adoption 


What are the greatest barriers to adopting 
the intervention? 

What supports will need to be in place for 
clinics to adopt the intervention? 


Perceptions of staff and leadership. 


Research team observational log. 
Post-intervention staff interviews. 
Post-intervention leadership interviews. 


Implementation 


What supports need to be in place to ensure 
consistent delivery of the intervention? 


Perceptions of staff and leadership. 


Post-intervention staff interviews. 
Post-intervention leadership interviews. 




What tools will be needed to deliver the 


Perceptions of research staff 


Training protocols for research staff. 




intervention consistently? 








What does the intervention cost? 


Cost data. 


Records of incentives awarded, costs 
of intervention supplies and staff time 
required. 


Maintenance 


What resources will be needed to maintain 
the intervention in the long run? 

What adaptations will need to be made to 

intpinratpi the interx/entinn into rp>r\\ ilar Pirar"tir~pi? 
II HcLJ I ale LI Itr II llfcM VfcM 1UUI 1 IIILU IcLjUldl pidLLILc: 


Perceptions of leadership. 


Research team observational log. 
Post-intervention leadership interviews. 


PARIHS 








Evidence 


What are staff perceptions of the evidence 
supporting the intervention? 

What are their attitudes toward the intervention? 

Does the intervention fit with their current 
clinical practice? 

Does the intervention meet a perceived 
need of their patients? 


perceptions of staff and leadership. 


Organizational Readiness to Change 
Assessment (ORCA) Evidence Scales 


Context 


What are the characteristics of the culture in the clinic? 


Organizational readiness measure 


Organizational Readiness to Change 




What are the characteristics of the leadership of the clinic? 


collected from staff and leadership. 


Assessment (ORCA) Context Scales 




What resources are available to the clinic? 







""Effectiveness outcomes are reported elsewhere [19]. 

setting in which the proposed change is to be 
implemented" [23]. Given the importance of context in 
a number of implementation frameworks e.g., [27-29] 
and the fact that RE-AIM does not specifically address 
context, we felt that assessing this element would add 
important information to our process evaluation. 
Specifically, it could enhance our interpretations of 
setting-related barriers and facilitators to this innovation. 

PARIHS s facilitation element is defined as an appointed 
role wherein an individual helps and enables others to 
change [23]. Evaluation of facilitation was not felt to be 
relevant to this study as the research staff administered 
the study's clinical intervention and were not providing 
such help or support to assist clinical staff with real-time 



implementation. However, information gained from asses- 
sing the RE- AIM framework, as well as PARIHS' evidence 
and context elements, was expected to provide sufficient 
information to inform development of future facilitation 
strategies. 

Data collection tools 

Data tools included pre-existing scales, surveys and in- 
terviews designed specifically for this study, observa- 
tional notes, data forms completed by research staff, and 
chart reviews to collect administrative data (Table 2). 

Chart review data 

In order to track Reach of the intervention, a screen- 
ing database was maintained which contained basic 



Hagedorn et al. Addiction Science & Clinical Practice 2014, 9:12 
http://www.ascpjournal.Org/content/9/1 /1 2 



Page 5 of 15 



demographic and diagnostic information on every patient 
that was approached regarding participation in the study. 
This information was used to determine the percentage of 
patients approached that agreed to participate and to 
compare basic demographic and diagnostic characteristics 
of those agreeing and declining participation. 

Cost records 

One consideration in regards to Implementation of a 
new intervention is the cost associated with the inter- 
vention. A standardized computer data entry screen was 
used to document the amount of time required for each 
participant intervention appointment and the monetary 
value of any vouchers earned. Staff time was converted 
into costs using three different wage rates plus a 30% 
fringe rate. Average wage rates were calculated for: 1) li- 
censed practical nurses, which clinic leadership indicated 
would be the most likely staff position to administer the 
intervention if one person were assigned to administer it 
to all patients, 2) the actual research staff that adminis- 
tered the intervention for the study (Bachelors level re- 
search assistants), and 3) licensed clinical social workers, 
which clinic leadership indicated would be the most 
likely staff position to administer the intervention if each 
case manager were assigned to administer it to their 
own patients. 

Patient surveys 

To collect information that could improve the Reach of 
the incentive intervention, participating patients com- 
pleted post-intervention surveys at the end of the eight 
week intervention period. The survey queried patients 
about what they liked and did not like about the inter- 
vention, any interactions they had with their clinical 
treatment team related to their participation in the inter- 
vention, and their thoughts on potential modifications to 
the incentive intervention. 

The Organizational Readiness to Change Assessment 
(ORCA) 

The ORCA is a 77-item scale designed to measure ele- 
ments of the PARIHS framework [30,31]. Items pertaining 
to the evidence and context elements were administered. 

The evidence scale both assesses the respondents' opin- 
ions regarding support for the proposed intervention 
based on research evidence, clinical experience and pa- 
tient preferences, as well as their perception of colleagues' 
related support. Because preliminary discussions with staff 
indicated that incentive interventions can be controversial, 
we added an additional eight items to assess personal 
opinions and intentions regarding incentive interventions 
(see Table 3). The context scale contains six subscales: 
leadership culture, staff culture, leadership practices, 
measurement (e.g., leadership feedback), readiness to 



Table 3 Items added to organizational readiness to 
change scale to assess providers personal preferences 
and intentions 

Construct Item 

Preferences • If monetary rewards can help patients to 
achieve abstinence, I support it. 

• Giving patients monetary rewards for staying 
abstinent or attending treatment is wrong, 
(reverse coded) 

• Patients who really want to stay abstinent 
shouldn't need monetary rewards, (reverse coded) 

• We need to use every possible tool, including 
monetary rewards, to help patients succeed 
in treatment. 

• Giving patients monetary rewards for abstinence or 
treatment attendance is an inappropriate use of 
VHA funds, (reverse coded) 

• Giving patients monetary rewards will recognize 
their achievements and give them a sense 

of accomplishment. 

Intentions • I would like to see the use of incentive interventions 
in this clinic. 

• There are specific plans in place to implement an 
incentive intervention in this clinic. 



change among opinion leaders and resources to support 
general practice changes. 

ORCA surveys were handed out at clinical staff team 
meetings after all patient participants had completed the 
intervention phase of the trial. No identifying informa- 
tion was collected so that completion of the assessment 
was voluntary and anonymous. 

Staff interviews 

To collect information on Adoption, Implementation and 
Maintenance, after all patient participants had completed 
the intervention phase of the trial, staff members that 
agreed to be interviewed were asked their opinions regard- 
ing the intervention and how it impacted the clinic, their 
interactions with their patients, and their workload. They 
were also asked about barriers and facilitators to maintain- 
ing the intervention as part of standard clinical care. No 
identifying information was collected during the inter- 
views other than the staff members' role in the clinic. 

Leadership interviews 

To collect information on Adoption, Implementation and 
Maintenance, after all patient participants had completed 
the intervention phase of the trial, clinic leaders were 
asked about their plans to continue the incentive interven- 
tion in their clinic, perceived incentives for maintaining 
the intervention, barriers and facilitators to maintaining 
the interventions, and types of support that would be most 
helpful in adopting and maintaining the intervention. 
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Research staff observational log 

To supplement information regarding Adoption, Imple- 
mentation and Maintenance, a shared research staff ob- 
servational log was created to document interactions with 
clinical staff and patients that provided information on 
their expressed attitudes toward the intervention and 
their comments on perceived barriers and facilitators 
to adoption, implementation, and maintenance of the 
intervention. The idea behind the log was that opinions 
might surface during day-to-day interactions with re- 
search staff that might not be readily reported when 
utilizing more formal data collection methods such as 
surveys and interviews. 



Data analysis 

Descriptive quantitative results are reported in percent- 
ages. Chi-square tests, and Z or t-tests were used to com- 
pare distributions, proportions and means of different 
subgroups. Qualitative information from open-ended sur- 
vey questions, interviews, and research log entries were 
transcribed and analyzed using common coding tech- 
niques for qualitative data [32]. The principal investigator 
(HH) and the research coordinator (E. Marut) independ- 
ently reviewed the transcripts and developed initial coding 
lists. A deductive analysis approach was used during 
the development of the initial coding list to focus codes 
on pre-determined domains of interest. This included 
attitudes toward the intervention, barriers to and faci- 
litators of implementation, and recommendations for 
improvement of the intervention. Following develop- 
ment and refinement of the initial coding list, each 
transcript was coded, with additional inductive codes 
added that identified important themes not represented 
by the pre-determined domains. Consensus meetings 
were held for review of consistency in coding. Inconsist- 
encies were resolved through mutual discussion and 
agreement. 



Results 

Patient screening data 

Figure 1 provides recruitment information including per- 
cent of contacted patients that participated and declined 
and reasons for declining. Table 4 provides demograph- 
ics for those that declined participation and those that 
agreed to a baseline appointment. African Americans and 
patients with stimulant dependence diagnoses agreed to 
participate at higher rates than patients in other demo- 
graphic and diagnostic categories. 

Costs 

Table 5 presents cost information. Most substance use 
disorders clinics have a breathalyzer unit available so the 
cost of the unit was not included as a cost of the inter- 
vention. However, if a clinic needs to purchase a breath- 
alyzer, one can be purchased for less than $500. The 
system to track appointment length indicated that mean 
appointment length was 7.0 minutes (SD = 4.5). Costs 
were assessed based on 15 minute appointments because 
the minimum appointment scheduling time in the VHA 
is 15 minutes and 15 minutes per appointment would 
have allowed sufficient time for 96% of all appointments. 
Staff costs were multiplied by the 16 scheduled appoint- 
ments rather than the number of appointments actually 
attended. This accounted for staff time spent document- 
ing no-shows and contacting patients to reschedule. 

Patient surveys 

The post-intervention survey was completed by 147 of 
the 165 patients (89%) that were assigned to the incentive 
intervention. When asked what they liked about partici- 
pating in the intervention, 94% of participants gave a 
response. When broadly categorized, the top response 
categories were "liked earning the rewards" (44.3%), "the 
intervention supported abstinence" (37.0%), and "liked 
working with the research staff (22.4%). One partici- 
pant summarized her experience in the following way: 



551 Eligible Patients Contacted 



369 (67%) Participants Scheduled for 
Consent/Baseline Appt 



332 (60%) Participants 
Consented 



37 (7%) Not consented 
25 (68%) No-shows 
12 (32%) Refused consent 



Figure 1 Participant recruitment. 



182 (33%) Not Interested 
109 (60%) No specific reason provided 
42 (23%) Not enough time to participate 
16 (9%) No reliable transportation 
11 (6%) Appointments are too frequent 
4 (2%) Not enough compensation offered 
4 (2%) Do not participate in research 
2 (1%) Too sick to participate 
1 (0.5%) Dropping out of SUD treatment 
(add to >100% ; some patients gave 
multiple reasons) 
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Table 4 Demographic characteristics of patients that declined participation and patients that scheduled baseline 



appointment 



Characteristic 


Declined participation 
N = 182 (33.03%) 


Scheduled 

N = 369 (66.97%) 


Male 


1 78 (97.80%) 


361 (97.83%) 


Female 


4 (2.20%) 


8 (2.17%) 


African American 


34 (18.68%) 


136 (36.68%) 


White 


124 (68.13%) 


194 (52.57%) 


Other non-white 


4 (2.20%) 


10 (2.71%) 


Age in years, median 


52.5 


50 


Psychiatric Diagnoses 






Depression 


59 (32.42%) 


131 (35.50%) 


PTSD 


29 (15.93%) 


46 (12.47%) 


Other anxiety, mood, or psychotic diagnosis 


37 (20.33%) 


53 (14.36%) 


Substance Dependence Diagnoses 






Alcohol Dependence 


172 (94.51%) 


307 (83.20%) 


Stimulant Dependence 


32 (17.58%) 


156 (42.28%) 


Other Substance Dependence 


18 (9.90%) 


33 (8.94%) 



"It was very positive. The program doesn't dwell on 
your short-comings. It is focused on positives with the 
incentives. It helped me to stay sober and helped me 
to commit to not drinking. I really liked it." 

When asked what they did not like about participating 
in the intervention, 78% of participants stated "nothing." 
The most frequent category, reported by 10% of partici- 
pants, were complaints about the reward system, e.g., the 
rewards were not big enough. Participants stated that: 

"I didn't like the reward scale, the odds of not winning." 

"Those "Good Job" things, I didn't like picking those." 



Table 5 Costs of the intervention 


Expense item 


Cost (Mean) 


Incentive Vouchers 


$103.07 


Rapid Urine Test Cups 


$60.90 


($5.25/cup x 11. 6 a ) 




Breathalyzer Mouthpieces 


$2.78 


($0.24 x 1 1 .6) 




Staff Costs 


LPN b = $102.72 


(16 appointments x 15 minutes each) 


BA C = $125.60 




LCSW d = $152.00 


Total Cost per Patient 


$266.93 - $316.21 



a 11.6 = mean number of intervention appointments attended, 
licensed Practical Nurse: $41,101 base + 30% fringe. 
c Bachelor's level research assistant: $50,275 base + 30% fringe. 
d Licensed Clinical Social Worker: $60,827 base + 30% fringe. 



"Dang it, (name of other participant) won $80 and not 
me!" 

Other responses included "the appointments were in- 
convenient" (8%) and "did not like the testing procedures" 
(3%). 

When asked to rate the impact of the incentive interven- 
tion on their personal treatment outcomes, 81 (55.1%) par- 
ticipants rated the impact as highly positive, 47 (32.0%) 
participants rated the impact as somewhat positive and 19 
(12.9%) participants rated the incentives as having no im- 
pact on their treatment outcomes. None of the participants 
rated the impact of the incentives on their personal treat- 
ment outcomes as either somewhat or highly negative. 

Only 22.8% of patients reported that someone from 
the clinical team had discussed their intervention partici- 
pation with them. Of these conversations, the majority 
were neutral (63.0%) and involved a clinical staff person 
either mentioning the opportunity to be involved in the 
study or asking how participation was going. The remain- 
der of the conversations were positive (37.0%) with the 
clinical team member indicating that it was a good thing 
for the patients recovery to be involved in the interven- 
tion. There were no reports from patients of clinical staff 
making negative comments about their intervention par- 
ticipation. There were only two instances where a patient 
reported that a clinical team member discussed specific 
urine and breathalyzer screen results from the interven- 
tion with the patient. 

Participants indicated that the intervention would have 
been more effective if it had been longer (greater than 
8 weeks), if the prizes available had been bigger, and if it 
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offered a larger number of prizes and more large and 
jumbo prizes. The participants indicated that the inter- 
vention would have been less effective if it had been 
shorter (less than 8 weeks), if the appointments had 
been less frequent (less than 2 times per week), and if 
the intervention had taken place in a group setting 
where patients would know each other s test results and 
reward earnings. 

Organizational readiness to change assessment 

The ORCA was completed by 13 staff members from each 
of the two participating substance use disorders clinics 
(Total N = 26/39, 67% response). At the first site, respon- 
dents included five social workers, three nurses, two psy- 
chiatrists, one psychologist, and two staff members that 
did not record their role in the clinic. At the second site, 
respondents included five social workers, one nurse, three 
psychiatrists, three psychologists, and one staff member 
that did not record his/her role in the clinic. 

Staff at both sites expressed agreement with the evi- 
dence supporting incentive interventions. On a scale of 
1 to 5, with 5 representing strong agreement, the mean 
evidence scale rating was 3.7 for site one and 3.9 for site 
two. Mean scores on the three evidence subscales (re- 
search evidence, clinical experience, and patient prefer- 
ences) were all above 3.7 for both sites. None of the 
differences between sites were significant. On the six 
items related to personal opinions regarding incentive 
interventions, site one had a significantly lower rating 
(M = 3.5) compared to site two (M = 4.2; p = .01) al- 
though site one was still in the supportive range (> 3). In 
regards to the intentions questions, on the item asking 
whether they would like to see an incentive intervention 
continue in their clinic, site one had a significantly lower 
rating (M = 3.8) than site two (M = 4.5; p = .027) but 
both sites were in the supportive range. In looking at 
individual scores, site one had three individuals that had 
clearly negative opinions regarding the incentive in- 
tervention which pulled down the overall score for the 
clinic despite the majority of staff expressing support. 
When asked if there were any specific plans in place to 
continue an incentive intervention in the clinic, the 
mean rating for both sites was 2.8 indicating that while 
they were supportive of the intervention, they were un- 
aware of any clear plans in place to sustain it following 
the research study. 

Staff at both sites also indicated that their sites had 
contexts that were supportive of innovation in general. 
The mean context scale rating was 3.6 for site one and 
3.3 for site two. Mean scores on five of the six context 
subscales (leadership culture, staff culture, leadership 
practices, measurement, and opinion leaders) were all 
above 3.1 for both sites. None of the differences between 
sites were significant. The only context subscale to have 



negative mean ratings from both sites was the resource 
subscale indicating that staff at both sites did not feel 
that they had the financial resources, training, facilities 
or staffing necessary to adequately support implementa- 
tion of innovations. Site two had a significantly lower 
mean rating on resources (M = 2.0) than site one (M = 
2.9; p = .01). 

Staff interviews 

Seven interviews were conducted with staff from site 
one and two were conducted with staff from site two. 
The professional roles of those interviewed included 
two nurses, two social workers, one psychologist, one 
addiction therapist, and three whose roles were not 
identified. 

When staff were asked what they liked about having 
the incentive intervention in their clinic, the most com- 
mon themes to emerge, mentioned by three or more re- 
spondents, were that the intervention motivated patients 
to continue treatment, the intervention provided the pa- 
tients with tangible rewards for success, the intervention 
provided the patients with additional support, the inter- 
vention provided the staff with objective outcome data, 
and that the intervention was a positive/enjoyable ex- 
perience for patients. For example, one staff member 
summed up her experience with the intervention in the 
following way: 

"I liked it because it motivated the clients. There was 
a possibility for them to win something. Also, praise 
goes a long way. They come to (name of treatment 
program) beaten down and the incentives provided 
them with positive reinforcement." 

When staff was asked what they did not like about hav- 
ing the incentive intervention in their clinic, six of nine 
responded nothing. Two staff reported a concern that the 
intervention focused patients on external rather than in- 
ternal motivation for change as exemplified by the follow- 
ing quote: 

"Sometimes it seemed that the patients enrolled in the 
intervention were more superficial in their treatment. 
They had to stay sober for the rewards and money, 
not for deeper reasons." 

Two staff also reported a concern that the intervention 
could trigger gambling behavior. 

When asked whether the intervention had any impact 
on clinic functioning or atmosphere, the most common 
theme, mentioned by five respondents, was that the inter- 
vention provided an additional opportunity to support pa- 
tients' recovery, as exemplified in the following quote: 
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"The intervention added to the overall goal of 
working together as a team to help the patients. I 
think the patients got the sense that, as a clinic, we 
were committed to helping them to get and stay 
sober. The incentive study helped with that." 

None indicated a negative impact of the intervention. 
When asked whether the study intervention had any im- 
pact on individual respondents' workloads, six of nine 
respondents stated that there was no impact. Those that 
reported an impact reported co-signing incentive inter- 
vention appointment notes and having more contact with 
patients during the aftercare phase of treatment. All three 
agreed that this increased interaction improved treatment 
for the patients. As one staff member described it: 

"There were a lot of notes to cosign, although it was 
always optional. I like to have the information about 
who was showing up for intervention appointments 
and who was testing positive and negative." 

When asked if the intervention influenced their inter- 
actions with their patients, five of nine respondents 
indicated that it did not. Those that did indicate an in- 
fluence reported talking with their patients about their 
study earnings and adding ongoing participation in the 
intervention to patients' aftercare recovery plans. 

When asked if they would like to see the intervention 
continue in the clinic after the research study ended, seven 
of the nine respondents said yes. The remaining two 
stated that their opinion would depend on the results of 
the study. When asked about barriers to adopting the 
intervention as part of standard clinical treatment, the 
most common theme, cited by six respondents, was cost. 
An additional barrier mentioned by three respondents was 
allocating staff time to administer the intervention. As 
summed up by one staff person: 

"The time commitment would be a barrier. You need 
staff that would be in charge of this. The expense 
would also be a barrier. I see the lab costs as being 
the most expensive cost. Sending confirmations to the 
lab is very expensive but I think it's really important 
that we do that." 

When asked what would make it easier to adopt the 
intervention as part of standard clinical treatment, 
themes reported by three or more respondents were 
dedicated staff and soliciting buy-in from all staff in the 
clinic prior to implementation to decrease resistance. 
For example, one staff person stated: 

"Everybody needs to be on the same team. There needs 
to be staff buy-in from everyone involved. Another 



thing is that it should not be considered a secondary 
part of treatment. It should be an integral part." 

Leadership interviews 

All three leaders of the participating clinics indicated 
that they were interested in implementing an incentive 
intervention but that it was not currently feasible due to 
significant barriers. In the outpatient substance use dis- 
orders clinics, the main perceived barriers were as fol- 
lows: staff were already stretched beyond capacity, the 
need to either gain additional staff or reallocate staff 
time from other priorities, and the need to identify a 
funding source for both the incentives and the rapid 
urine test cups. As one of the leaders put it: 

"The biggest barrier is that I would have to pull staff 
from other tasks in order to do it. There are only so 
many staff members and something else would have 
to give." 

In the dual diagnosis partial hospitalization clinic, the 
main barrier cited was that their program was primarily 
group based and therefore, they would need expert con- 
sultation to determine the best way to incorporate the 
intervention into the group setting. Staff time was also a 
concern for this clinic. 

All of the leaders indicated that there were significant 
incentives for pursing implementation. The leaders in the 
substance use disorder treatment clinics noted that the 
intervention appeared to improve treatment retention and 
that this improved their clinics' functioning on a VHA 
substance use disorders treatment continuity of care per- 
formance measure that was monitored by national leader- 
ship as demonstrated by the following quote: 

"For me, the continuity of care is a big incentive. It 
(the intervention) keeps the patients engaged in 
treatment and coming in for appointments." 

The leader of the dual diagnosis clinic noted that incen- 
tive interventions are an empirically supported treatment 
and that empirically supported treatments generally, and 
incentive interventions specifically, were strongly sup- 
ported by national leadership. When asked what types of 
assistance would be most beneficial to support implemen- 
tation, suggestions included educating staff and facility 
leadership regarding the evidence for incentive interven- 
tions and expert consultation in designing an incentive 
intervention that would fit into the current practices of 
the clinic. 

Research observational log 

Research staff recorded over 150 contacts with clinical 
staff in the log. The vast majority of recorded contacts 
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were positive. Clinical staff demonstrated support for the 
intervention both through assisting the research staff 
with recruitment of participants and with maintaining 
contact with participants during enrollment in the study. 
Clinical staff comments indicated the intervention was 
useful to their work with patients by allowing regular 
monitoring of urine screen results and keeping patients 
connected to aftercare. When one of the research assis- 
tants approached a staff member about recruitment of a 
specific patient, the staff member spontaneously replied: 

"I really appreciate how the study keeps the patients 
coming back to the clinic. It helps keep them 
connected to aftercare after they complete their 
intensive program. It gives them another long term 
connection and support." 

The research log also prompted the research staff 
to identify and act on suggestions from clinicians to 
enhance clinician involvement with the intervention. For 
example, staff at both facilities requested that they re- 
ceive regular updates on participants' progress in the 
incentive intervention. Research staff worked with the 
clinicians to identify the best way to achieve this goal at 
each facility. A second suggestion that came from clin- 
ical staff was that the patient s case manager should be 
invited to the final intervention appointment in order to 
celebrate this achievement with their patient; however 
when this was offered, staff attended infrequently. 

As stated above, the vast majority of recorded interac- 
tions were positive but there were barriers and negative 
responses to the intervention recorded as well. The 
major barrier to integrating the intervention into clinic 
processes at one facility was that the clinicians did not 
want to rely on the rapid urine test cups for clinical test- 
ing of substance use. They preferred to send samples 
directly to the laboratory. This created a great incon- 
venience for patients who were often asked to submit 
two urine samples (one for research and one for clinical 
purposes) within a very short time frame. During the 
course of the intervention, several instances arose where 
clinical staff required immediate knowledge regarding 
whether a patient was under the influence of substances 
and sought out research staff to request access to a rapid 
test cup, which the research staff readily provided. As 
the clinicians gained familiarity with the test cups, they 
gained confidence in the rapid test cup results and even- 
tually allowed patients to use their research sample as 
their clinical sample. An additional complaint was that 
some clinicians noted that patients were consistently 
attending their research intervention appointments but 
skipping their clinical care appointments. 

Research staff recorded over 50 interactions with patients 
involving spontaneous comments about the intervention in 



the research log. The majority of these comments were 
positive and related to patients' perceived benefits of the 
intervention. Several patients reported that the inter- 
vention was helping them to maintain their sobriety. 
Four patients recounted specific instances in which they 
were experiencing a craving to use substances and thought 
about the incentives they hoped to earn to assist them in 
fighting the craving. In one of these cases, the participant 
stated while he was picking that he had really wanted to 
have a drink the previous weekend but that he did not 
want to have his picks reset. He then pulled out a $20 
voucher and was so excited that he started to dance 
around the office. He stated: 

"That reward was so worth not having a drink over 
the weekend!" 

Several participants also reported that they had recom- 
mended participation in the intervention to other patients. 
Additional positive comments included appreciating being 
able to see their urine test cup results immediately and 
that the intervention gave them a documented record of 
urine screens that their clinicians could access. Patients 
also expressed satisfaction with the wide selection of 
items available at the VHA gift shop that could be pur- 
chased with their vouchers. The majority of the complaints 
about the intervention centered on disappointment with 
not drawing the larger vouchers. 

Discussion 

The final step in conducting our process evaluation was 
to triangulate the results from the various data sources 
to answer the questions that were developed based on 
the elements of the selected frameworks (see Table 1) 
and to summarize lessons learned (see Table 6). 

Reach 

Screening data indicated that 60% of potential participants 
agreed to enroll in the intervention. The most common 
reasons for declining to participate related to the time 
commitment involved in having to attend appointments 
twice per week. This would suggest that the intervention 
may be most appropriately targeted to patients that will be 
coming to the clinic two or more days per week for other 
treatment services so as to reduce trips to the facility 
solely for the purpose of the 15 minute intervention ap- 
pointment. African American patients and patients with 
stimulant dependence diagnoses agreed to participate at 
higher rates than patients in other demographic and diag- 
nostic categories. The reliability of this observation as well 
as its clinical significance would need to be assessed in fu- 
ture research. It is clear from the patient survey and re- 
search log data that the vast majority of patients that 
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Table 6 Suggestions to enhance implementation efforts 
based on the RE-AIM framework 

Reach • Target intervention to patients that will be 

attending treatment at least twice per week for 
other treatment services. 

Effectiveness • Share results of VA and non-VA trials 

Adoption • Solicit explicit support from the highest levels of 

the organization through, for example, performance 
measures or treatment recommendations. 

• Identify or create measures of clinic effectiveness 
which can be utilized to identify gaps in performance 
and monitor the impact of implementation. 

• Solicit agreement in advance for designated funding. 

• Educate leadership about time commitment related 
to the intervention and potential strategies for 
integrating the intervention into current practices. 

• Adopt incrementally. Start with a specific treatment 
track or clinic to reduce staff and funding burden 
until local evidence of effectiveness and feasibility is 
available to support spread 

Implementation • Disseminate information to educate staff about 
the process, goals, and value/effectiveness of the 
intervention and engage them in planning for the 
intervention from the start. 

• Provide expert consultation on how to adapt the 
intervention for specific clinic environments. 

• Train staff on urine test cups and breathalyzer 
including sensitivity and specificity of the screen 
results. 

• Make scripts available for communicating positive 
and negative test results to patients. 

• Supply a tracking database to ensure consistency 
in awarding prize picks. 

• Provide a step by step intervention appointment 
protocol. 

• Facilitate documentation in the electronic health 
record. 

Maintenance • Ensure all staff is aware of their responsibilities related 
to incorporating information from the intervention 
into clinical interactions with patients to facilitate 
integration into the clinic. 

• Consider option of having case managers administer 
the intervention to their own patients rather than 
having one or two individuals responsible for the 
intervention. 



agreed to participate felt the intervention experience was 
a positive one. 

Effectiveness 

The primary outcomes of this study are reported else- 
where [19]. However, in brief, in the alcohol dependent 
subgroup, incentive participants submitted significantly 
more negative samples (13 vs. 11 samples), were retained 
significantly longer (7 vs. 6 weeks), and achieved signifi- 
cantly longer median durations of abstinence (16 vs. 9 
consecutive visits compared to usual care participants). 



Intervention effects were non-significant for the stimu- 
lant dependent subgroup. While these non-significant re- 
sults for the stimulant dependent group were surprising, 
we believe these results were related to the characteristics 
of the recruited participants, specifically that they ap- 
peared to have less serious stimulant dependence com- 
pared to samples in several previous significant trials. 
Despite the non-significant results of this one trial, we be- 
lieve that the extensive literature supporting incentive in- 
terventions with stimulant dependent patients supports 
implementation e.g., [33-36]; and further the results of this 
trial may provide some guidance in selecting patients most 
likely to benefit from incentive interventions. 

Adoption 

Leadership identified some strong rationales for adoption 
of incentive programs. They perceived that incentive in- 
terventions would be valuable to improve their treat- 
ment retention performance ratings. Additionally, they 
believed national leadership viewed implementation of 
incentive interventions as a priority, and in fact, incen- 
tive interventions had been designated by national lead- 
ership as an evidence-based practice. The fact that the 
clinic leaders both expressed dissatisfaction with their 
own clinics' performance on the treatment retention 
performance rating and perceived the incentive inter- 
vention as directly addressing this issue appeared to be 
the strongest motivator for adopting a change in prac- 
tice. Clinic leaders that do not perceive issues with the 
effectiveness of their program at baseline may be less 
likely to express motivation to adopt practice change. 
This highlights the importance of having measures 
available to assess clinic effectiveness which can help to 
identify gaps in performance and assess the impact of 
changes in clinic practices. 

However, these strong rationales for adoption could not 
override the greatest barrier, which was lack of resources 
in the form of staff time and funding for vouchers and 
urine testing supplies. Multiple data sources, including the 
ORCA, staff interviews, leadership interviews, and the re- 
search log, all indicated that lack of resources was the pri- 
mary perceived barrier to adoption of the intervention. To 
address the perceptions regarding staff time, we can sug- 
gest two implementation strategies that may decrease 
leadership concerns about staffing burden: providing edu- 
cation in advance as regards the amount of time necessary 
to administer the incentive intervention (15 minutes or 
less per instance); and presenting various options for inte- 
grating the intervention into current clinic staffing pat- 
terns especially when group sessions are the predominant 
modality. While cost of the intervention is traditionally 
considered to fall under the Implementation construct of 
the RE-AIM framework, we consider cost under the 
Adoption construct given the high relevance of funding 
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concerns to the decision whether or not to adopt this par- 
ticular intervention. The mean total cost of the eight week 
intervention was $266.93 - $316.21 per patient depending 
on the salary rate of the individual administering the inter- 
vention. This cost could be reduced by using lower wage 
staffing. While a full cost-effectiveness evaluation is be- 
yond the scope of this paper, the incentive intervention 
was shown to increase participants' longest duration of 
abstinence during treatment [19] which is one of the best 
predictors of long-term patient outcomes [37-39]. Olm- 
stead and colleagues have published two articles reporting 
on the cost-effectiveness of incentive interventions [40,41]. 
The potential for decreased societal costs related to med- 
ical expenses, criminal/legal costs, and lost productivity 
would suggest that this is a low price for an intervention 
that improves substance use disorders treatment outcomes. 
Providing education regarding the low cost and the cost- 
effectiveness of the intervention and soliciting agreement 
in advance from facility leadership for dedicated funding 
appear to be essential to adoption. 

It should be noted that adoption of many new evidence- 
based practices into an existing clinic milieu will require 
additional resources, so such barriers are not unique to in- 
centive interventions. Another potential implementation 
strategy may be to assist clinic leadership in identifying 
current clinic practices that are not evidence-based or that 
have been shown to contribute little to treatment out- 
comes, e.g., didactic education [42]. The implementation 
team and the clinic leadership could then focus on strat- 
egies for reallocating existing resources from current non- 
evidence based practices to implementation of evidence 
based practices not currently available in the clinic. 

Clinic leadership also indicated that it would be an 
easier "sell" to facility leadership to begin by targeting 
the intervention to a small group of patients. This would 
offer a way to test the intervention in the clinic and col- 
lect local outcome data that could then be used to advo- 
cate for sustained funding support for the intervention 
at a later date. Starting with a "test" of the intervention 
could also lessen staff resistance and allow them to ob- 
serve the effectiveness of the intervention in their own 
clinic and the impact of the intervention on their work- 
load prior to committing to sustained implementation. 

Implementation 

Once a clinic has made a decision to adopt the incentive 
intervention, supports must be in place to ensure that 
implementation is carried out with high levels of fidelity 
and consistency. The staff and leadership interviews sug- 
gested several ways that high quality implementation 
could be enhanced. These include: 1) involving staff in 
implementation planning to increase knowledge regard- 
ing essential intervention elements; 2) providing support 
for adapting the intervention to fit the current clinic 



structure through expert consultation and recommended 
strategies based on common clinic structures; and 3) de- 
fining clear roles for all clinical staff related to the inter- 
vention so that it would be a clinic-wide initiative rather 
than viewed as only the responsibility of specific people. 
In addition to these supports, the research log suggested 
that it would be beneficial to train all staff on the use of 
rapid urine test cups and their specificity and sensitivity 
prior to implementation to address concerns about the 
reliability of that technology. 

Other observations related to implementation tools 
that could be used to enhance consistent delivery of this 
intervention come from our experience in training the 
research staff. Research assistants (RAs) were provided 
with training on the accurate use of the rapid urine test 
cups and the breathalyzer unit and with scripts for how 
to communicate screening results to patients. Addition- 
ally, a database was created that tracked the number of 
picks that a patient was entitled to at a given appoint- 
ment based on their previous record of attendance and 
test results. This simplified the work of the RAs and en- 
sured the consistency with which the incentives were 
distributed. RAs were also provided with a manual that 
described the steps of an intervention appointment from 
beginning to end. Once an RA had completed their 
training and read the manual, they completed a mock 
intervention appointment with one of the investigators 
(HH, DK) prior to completing intervention appoint- 
ments with patients. These same training tools and steps 
could be provided to clinical staff to ensure consistent 
delivery of the intervention. 

Maintenance 

In examining Maintenance, we were primarily interested 
in adaptations that could be made to the incentive inter- 
vention that would enhance the probability that the 
intervention would be sustained. As reflected in the bar- 
riers to adoption, dedicated staff time and funding would 
be essential to maintenance and should be secured prior 
to a decision to adopt the intervention. In addition, both 
staff and leadership interviews indicated that fully inte- 
grating the intervention into the clinic would be essential 
to maintenance. The intervention, as delivered during the 
study, was viewed as a stand-alone intervention that was 
the responsibility of specific individuals (in this case, re- 
search staff). While some individual staff members did re- 
port utilizing information about patients' progress in the 
intervention in their therapeutic interactions, the patient 
survey information would suggest that this was infrequent. 
It does not seem that clinicians were uninterested or op- 
posed to being more involved, as staff at both clinics pro- 
vided unsolicited suggestions regarding ways that clinical 
staff could be kept more up-to-date on patient progress in 
the intervention. Therefore, it is more likely that they did 
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not know enough about the intervention or how best to 
incorporate information from the intervention into their 
therapeutic interactions. This could be mitigated by 
providing better education to all staff about the inter- 
vention and by developing explicit expectations regard- 
ing how staff should integrate information from the 
intervention into their interactions with patients. An- 
other option would be to have each case manager pro- 
vide the incentive intervention to their case load rather 
than assigning administration of the incentive interven- 
tion to one or two isolated individuals. This option 
would also eliminate the staff concern that patients were 
attending their intervention appointments more regu- 
larly then their clinical appointments as these would be 
provided concurrently. 

Evidence 

Ratings on the ORCA indicated that staff at both facil- 
ities generally agreed with the evidence supporting the 
intervention, that the intervention met the needs of their 
patients, and that the intervention fit with their treat- 
ment philosophy. Staff interviews supported results from 
the ORCA. The research log data also supported staffs 
reported enthusiasm for the intervention with multiple 
reports of clinical staff members assisting research staff 
with recruitment and facilitating communication be- 
tween research staff and study participants. 

However, despite this general enthusiasm, there were 
clear exceptions to this support with a small minority of 
staff reporting a serious conflict between their personal 
treatment philosophy and incentive interventions. They 
strongly felt that the incentives created a "false", external 
motivation for abstinence that would undermine patients' 
internal motivation for change and would interfere with 
longer term treatment success. It is likely that these beliefs 
are even more prevalent in community treatment settings 
that do not have the strong focus on evidence-based prac- 
tice currently promoted within the VHA system. While 
these beliefs tend to be highly entrenched, having discus- 
sions up front with staff and leadership to identify such 
beliefs or concerns and to determine what types of evi- 
dence staff would find compelling might encourage the 
staff to consider supporting implementation. For example, 
staff may simply not be aware of the level and detail of evi- 
dence supporting incentive interventions. They also may 
benefit from a more in-depth understanding of how be- 
havioral reinforcements function in a variety of settings. 
They may be encouraged by discussing incentive interven- 
tions with other clinics that have already implemented or, 
as suggested earlier, they may be willing to engage in a 
brief test of incentive interventions in their own clinic that 
would allow staff to see the benefits and risks of the 
intervention among their own patients. The important 
point is to engage the staff in a respectful discussion 



about implementation early on so that their concerns 
can be heard and addressed prior to a formal decision 
to make a change in how they are going to treat their 
patients. The Promoting Awareness of Motivational In- 
centives (PAMI) website, based on knowledge gained 
from the National Institute of Drug Abuse (NIDA) Clin- 
ical Trials Network (CTN) Motivational Incentives for 
Enhanced Drug Abuse Recovery study, provides several 
excellent resources for staff education and training in- 
cluding testimonial videos from patients and clinicians 
regarding the impact of incentive interventions on clinic 
culture and patient outcomes (http://www.bettertxoutcomes. 
org/bettertxoutcomes/PAMI.html). 

Context 

ORCA survey results indicated that staff perceived a gen- 
erally positive leadership/staff culture and positive leader- 
ship practices. The ORCA survey supported interview 
data, indicating that staff generally disagreed that they had 
access to the necessary resources to support innovation. 

Despite general agreement with the evidence support- 
ing incentive interventions and contexts that support 
innovation in general, the clinics involved in this study 
perceived substantial barriers to implementation. The 
implication is that barriers may be perceived as even 
more substantial in sites that do not have these support- 
ive characteristics. This highlights the importance of 
conducting local assessment of Evidence and Context 
when undertaking implementation efforts [43]. 

Limitations 

One limitation of this study is that the ORCA was only 
collected after the intervention had been running in the 
clinic for 18 months. It would have been ideal to have 
also collected this information prior to the start of the 
intervention period, as experiencing the intervention 
functioning in their clinic may have altered staffs atti- 
tudes toward the intervention. Previous studies have in- 
dicated that providers who have prior experience with 
tangible incentive programs display more positive beliefs 
about incentive programs and lower ratings of barriers 
to adoption than those who report no experience with 
such programs [44-46]. These findings reinforce the sug- 
gestion, also made by other authors, that exposing treat- 
ment providers to incentive programs through a pilot or 
targeted demonstration may be a useful strategy for en- 
couraging dissemination [47]. A second limitation was 
that the sample size for the staff interviews was small, 
particularly in site two, and therefore may not ad- 
equately represent the opinions of the population of 
clinic staff at the sites. Clinic staff that agreed to partici- 
pate in the interviews may have had more positive opin- 
ions about the intervention compared to those that 
declined to be interviewed. Finally, because this was an 
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effectiveness trial with research staff conducting the incen- 
tive intervention rather than an implementation trial with 
the clinical staff conducting the intervention, staff percep- 
tion of the intervention, particularly related to its impact 
on workload, may be quite different than if they were re- 
sponsible for conducting the intervention themselves. 

Conclusion 

An implementation-focused process evaluation guided 
by the RE-AIM and PARIHS frameworks led to signifi- 
cant insights about the processes involved in implemen- 
tation of incentive interventions and allowed us to 
formulate concrete suggestions about implementation 
strategies. Some of the most important insights were: 1) 
the need to spend substantial effort on education and 
persuasion strategies in sites where staff are skeptical of 
the value of implementing incentive programs, and 2) 
the need to assist leadership in developing strategies for 
financing and staffing. Overall, the study demonstrates 
the value of conducting implementation process evalua- 
tions as a concurrent part of effectiveness research. 

The goals of this paper were to describe an implemen- 
tation-focused process evaluation conducted during a 
randomized trial and distil findings into practical guid- 
ance to others who wish to implement incentive inter- 
ventions in substance use disorders clinics. Table 6 
presents a summary of the key lessons learned from this 
effort. It is our hope that this summary speaks to the 
value of the effort expended on the process evaluation 
and highlights issues to consider when attempting im- 
plementation of an incentive intervention. 
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